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ABSTRACT 

A common concern of state*level and post secondary 
institutional decisionmaking is the limitations of the state planning 
and budgeting process as it relates to educational change and 
accountability! increased regulations, the cost of obtaining 
information, the difficulty of assessing outcomes, the inability to 
discourage nonproductive programs, and lack of information about 
benefits to the ta^payer^ Interest in performance- or 
outcome^oriented budgeting is increasing, and a number of states now 
have performance audit staffs, A variety of technigues are used by 
these staffs: outside consultants, institutional assessments, and 
combined audit and program review* NCHEMS has developed an Inventory 
of Higher Education Outcome variables and Measures to be used by 
decisionmakers in dealing with this problem. The most needed outcome 
measures were determined from two surveys of state^ level and 
institutional administrators* Pour major difficulties occur in 
collecting and using outcome informations (1) explicit measures of 
educational outcome have been hard to come by; (2) interpretation of 
information (to determine cause and effect) is difficult; (3) general 
goals often lack translation into specific objectives; and (4) the 
usual time span of a budget limits the assessment of higher education 
outcome s, (Author/MSB) 
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Introducing Higher Education Outcome Information 
Into the State Planning and Budgeting Process 

Over the last several years ^ I have engaged in a number of discussions with 
state-level and postsecondary institutional decisionmakers about the problems 
associated with the state planning and budgetary process. The discussions 
generally have started with a focus on our troubled Inflationary economy ^ energy 
and conservation problems ^ the influx of state agencies competing for the state 
tax dollar^ and the consequences of a decline or leveling off of college enroll- 
ments* Sooner or later, however, the discussions have zeroed in on the problems 
and limitations of the state planning and budgetary process and its impact on 
change. Innovation^ and accountability In postsecondary education. For example^ 
college administrators and faculty have expressed a growlrfgr concern about the 
increased regulations and controls being placed on the Institutions by the 
budget process. They have expressed equal concern about what they feel are 
Inappropriate and unreasonable demands from a wide range of state agencies for 
budget-related information that is costly to obtain. They also have indicated 
great frustrations about their inability to do an effective job of telli^ng their 
Institution's story in terms of the intended and actual outcomes (I.e., impacts, 
effectiveness, and products) resulting from the institution's programs. State- 
level officials and staff, on the other hand^ have registered major concerns and 
complaints about the inability of the budget process's inability to discourage the 
They have expressed concern about the process' inability to discourage the 
perpetration by higher education and other social agencies of outmoded and non- 
productive programs. Finally, they have continually pointed out the lack of 
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good information about the returns of benefits citizens in the state will get 
or are getting froni the tax dollars invested in postsecondary education. 

Ultimately s the discussions with both state-level and college officials have 
ended with an expressed need for a more effective system for planning and 
budgeting. From the institutional point of view, this expression of need has 
been qualified by the need for a system that recognizes the unique mission, 
objectives, and achievements of each individual institution and program, is 
cognizant of the -'state of the art" with respect to the identification, measure- 
ment, and evaluation of educational outcomes, and considers the costs for obtaining 
outcome information. From the state-level point of view, the call for a more 
effective planning and budgeting system has been accompanied by demands for a 
system that links the resources used to the results produced, cuts out waste and 
inefficiency, and encourages a higher quality of education without having to 
always increase the costs (Folger 1975), 

Central to the requests from both groups is an emphasis on incorporating infor- 
mation about the Intended and actual outcomes or performances of an institution 
and its programs into the state-level planning and budgeting process • How might 
such a porformance- or outcome-oriented planning and budget process be developed 
and implemented? Who should be involved? What are the implications or consequences 
of such a process for institutions and states? From my vantage point, one of the 
major goals of this seminar is to begin getting a better grasp of the answers 
to these questions. With this in mind, the purpose of the remainder of this 
paper is to provide some background with respect to these questions. I will: 
(1) provide a brief overview of some of the current practices with respect to 
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collecting higher education outcome Information for use at the state level, 
(2) suggest some of the different kinds of outcome Infonnjitlon valued at the 
state level, and (3) highlight some of the problems and issues that are likely^ 
to be encountered in moving In this direction. 

Current Practices 

As a result of the pressures mentioned above^ state legislatures and state 
executive offices have increased their interest in and demand for better 1nfor= 
mation about the performance and effectiveness of tax-supported programs* For 
example 5 Mark Chadwin^ who is Director of the Illinois Economic and Fiscal 
Commission^ has stated that, "In I97O5 no state legislature had a full-time 
staff responsible for evaluation of program effectiveness* Now^ over a dozen 
have committees, commissions, or auditor's offices functioning in this area^ and 
more such bodies seem tj appear almost monthly" (1975:45), Lyman Glenny, in 
■his recent study on state budgeting for higher education, reported that, of the 
17 states studied, have separate executive or legislative staffs that perform 
policy-oriented performance or evaluation audits^ as distinct from routine fiscal 
audits" (1975:15-16), He also pointed out that performance audits of higher 
education Institutions have been made by these separate audit staffs and the role 
these staffs will play in budgeting for higher education will probably grow in 
Importance. 

Exactly what these audit staffs do is described in various ways. In some places, 
these program evaluation activities are referred to as "program auditing,." In 
others, they are called "program review," "performance post-auditing^" "legislative 
oversights," or "effectiveness auditing." However, no matter what the terminology, 
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the intent is the same! to measure the performance of programs 1n terms of 
actual results and impacts. As this implies^ the specification of quantifiable 
objectives and the collection of data about the indicators linked to those 
objectives are activities central to these approaches. 

Attempts to implement these approaches to measuring institutional performance 
at the state level range considerably with respect to: 

1. Who is responsible, 

2. What techniques are useds and 

3. How programs are selected for review. 

Who is responsible? In some states, such as In Virginia, Wisconsin, Hawaii, 
and Montana, numerous program evaluation activities are conducted by the 
legislative auditor^s office. * In other states, such as in Illinois, New York, 
and Florida^ the state higher education board staff play major roles In 
program evaluation and review. Evidence of this trend Is the addition of new 
staff who have management science, program evaluation, and statistical analysis 
skills. In Nebraska* program evaluation teams are being given responslbn ity 
for, what Is referred to as, ''program post auditing," These evaluation teams, 
which are composed of persons from the institution, the lay community, and, in 
some instances, the state legislature, judge the efficacy of selected depart- 
ments based on criteria linked to the stated goals of those departments. 



*The extent to which these offices are independent from the legislature varies 
greatly. However, the trend, which is encouraged by the General Accounting 
Office ((5A0) guidelines and the accounting profession, is toward greater 
autonomy (Chadwin, 1975:45) - 
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What techniques are used? In New York, Florida, and New Jersey, outside 
consultants are brought. in to conduct intensive reviews of selected programs 
(e.g., high cost graduate programs). According to Berdahl (1975:23), the use of 
external consultants has considerable merit, but it definitely adds to the cost 
of evaluation. He dted, for example, that in a recent New York doctoral program 
evaluation, which employed outside consultants, the budget was estimated to be 
$59,000, 

In some states, such as Hawaii, Washington^ and Illinois, program evaluation 
places major emphasis on performance indicators collected from the institutions. 
State agencies in Hawaii, for example, have identified a select number of 
performance measures against which institutions can be audited and evaluated 
through "a variance reporting procedure" (Meisinger, Purves^ and Schmidtlein^ 
1975:98). However, difficulties In actually gathering this information for use 
1n budget requests have prevented the procedure from being totally Implemented* 

In Wisconsin, the University system employs an approach for measuring the per- 
formance of academic programs that Includes both a program audit and a program 
review* Three basic organizing principles are used in Implementing this approach 
(Smith, 1975), First, it is Important that the audit and review rest with the 
institution. Second, It is Important to distinguish between " program audit as 
a process of Identifying programs which for any of several reasons should be 
given intensive review and program review as the comprehensive and intensive 
examination of a particular program." The third organizing principle Is that 
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the process of audit and review should be an Integral part of mid- and long- 
range planning, rather than a reaction to crisis situations. 

How are programs selected for review? In several states, program audits of all 
programs are conducted annually using a limited set of performance indicators 
such as enrollment trend data, degrees produced, student-faculty ratios, and 
costs relative to disciplines. These audits are usually employed to monitor 
major changes in the indicators collected so that programs in potential trouble 
in terms of efficiency and productivity can be Identified for more intensive 
program review. An alternative to the annual audit of all programs is the use 
of a program review cycle in which just the programs In a given category are 
scheduled for Intensive review every three or five years. The Florida Board 
of Regents, for Instance, has proposed a program review of programs categorized 
by academic program clusters on a five-year cycle. 

West Virginia uses still another approach for selecting programs for review. 
Using criteria related to costs, outcomes, priorities, and program quality, 
each institution is asked to make "forced ratings" about each of their programs. 
Sixty percent of the programs are to be rated as normal on each of the four 
variables and twenty percent are to be placed on each side of the normal range. 
Each Institution is then asked to select which program it wishes to have reviewed 
in depth (reported by Berdahl , 1975). 

Berdahl (1975) has noted that potentially there 1s much peril in using quantitative 
indicators as the only means for selecting programs for intensive review. In - 
support of this cautionary note, he dted the following: 
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For a variety of dudgmental reasons, administrators or planning 
committees for a particular Institution might ask for more intensive 
review of a program which had not been selected through the audit 
process. For examplei the anticipated retirement or departure of 
key faculty members in a given program might create a presumption 
that review of the program should be undertaken prior to restaffing; 
or a regularly scheduled site visit by an accreditation team might 
catalyze an institutional audit and review of a program In conjunc- 
tion with the preparation of documents for the visitors, or planning 
studies concerning the minimum staffing which should be maintained 
for the essential programs of an institution might generate need for 
review of particular programs; or recommendations from system-wide 
or institutional task forces on curriculum changes could also 
generate need for program review. 



Outcome Indicators Valued at the State Level 

Whether you are Interested in measuring institutional performance from a state 
or an institutional perspective, one of the most difficult tasks Is trying to 
identify what outcomes or performances should be assessed. Not only is it 
difficult to determine just what are the intended or actual outcomes of an 
institution or a program, but It Is equally difficult trying to reach consensus 
about which outcomes are the most important to measure. One of the major efforts 
in the Outcomes of Postsecondary Education Project at the National Center for 
Higher Education Management Systems (NCHEMS) has been to develop products designed 
to help institutional and state-level decision makers deal with this problem* 



One of the initial products developed 1n the Outcomes Project to deal with this 
outcomes identification problem was the Inventory of Higher Education Outcome 
Variables and Measures (Micek and Wallhaus, 1973), Two major criteria were 
applied in constructing the Inventory, One was to develop a^ list of 
outcome lyarlables or constructs that would serve as many kinds and levels of 
decision makers as possible. Second, the Inventory was Intended to be as compre- 
hensive as possible in that It would cover not only instructional programs but 
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also major noninstructional areas such as research, community education, and public 

service. Application of these criteria led to the construction of an inventory 

with the following general structure: ^a^. 



Figure 1 

Outline of Major Categories. In NCHEMS 
Invintory of Outcome Variables and Measures 



1*0 Student Growth and Devalopment 

1,1,0 Knowlidgi and Skills Davalopfrent 
1,1*1*00 Knowledge Davalopment 
1.1,2,00 Skills Davalopment 

1,1.3*00 Knowladga and Skills Attitudes, Values, and Beliafs 

1,2.0 Soctal Devalopmint 

1,2.1*00 Social Skills 

1*2*2.00 Social Attitudes, Valuas, and Beliefs 

1,3*0 Parsonal Development 

1.3.1.00 Studant Health 

1*3.2*00 per.snnal Attitudes, Values, and Ballafs 

1.4,0, Career Davalopment 

1.4.1*00 Career Preparation 

1*4.2,00 Career Attltudas, Values, and Beliefs 

2*0 Davelopment of New Knowledge and Art Forms 

3.0 Community Davalopment and Servica 

3,1,0 Community Devalopment 

3.2,0 Conmunity Servici 

3*3.0 Longtr-Tarm Coraiunlty Effects 
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Within each area of the Inventory, general outconie variables have been defined, 
toyether with specific potential measures or indicators of those variables. 
Thus, for example, such variables as "development of general knowledge," "develop- 
ment of specialized knowledgei" and "development of critical thinking and reasoning 
skills" have been identified with the knowledge and skills development area 
(Category 1.1,0). Measures that might be used to assess these variables include 
student scores on various standardized tests, numbers of graduates accepting 
employment in their major field of study as a percentage of total graduates in 
that field, and average student and/or former student scores measuring their 
degree of satisfaction with their ability to apply what they know.* 

As the Outcome Project staff worked on the Outcomes Inventory and the problems 
related to obtaining and using outcome information, a major question of concern 
was (and continues to be) "What are specific outcomes and associated indicators 
that are valued and needed by different Institutional and state-level decision 
makers?" It was believed that an answer to this question would provide a 
better understanding among Institutional and state-level officials about what 
outcomes are commonly endorsed by both groups and those the are of unique 
interest to persons in each particular group. Second, 1t was felt that such 
an understanding might enhance the dialogue between these different parties. 
Finally, If a set of high priority outcome measures could be identified, 
then work could begin In identifying and developing procedures for documenting 
and evaluating those outcomes. 



*A complete discussion of the development of the Inventory and its possible use 
is given In An Introduction tP the Identification and Use of Higher Education 
Outcome Information (Micek and Wallhaus, 1973). 
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In pursuit of an answer to the question stated abovfit two studies were conducted 
to Identify the outcome information instituttonal administrators (including 
presidents and top-level administrators for academic planning^ student affairs, 
and budget and finance) and state-level officials and staff (including governors, 
legislators, state budget officers, legislative staff, and state directors of 
higher education boards or coordinating bodies) need for their decision-making 
responsibilities concerning higher educational programs. 

The survey questionnaire used in the two studies was based on the outcome 
categories and measures contained in the Outcomes Inventory previously mentioned 
and was designed to: (1) identify the outcome information areas decision maker^s 
feel are most important (see Figure 2), and (2) identify the extent to which decision 
makers "need to know" and "have access to" specific outcome measure s. For 
example, in Outcome Information Area D, which contains a set of outcome measures 
related to Student Occupational Career Development, a decision maker had the 
opportunity of indicating his perceived need (or lack of it) for each of 13 
specific measures (see Figure 3), 

Various descriptive analyses of the data obtained from the surveys have been 
conducted and the results of these analyses are available from the Center, 
Figure 4 presents a list of twenty outcome measures that 601 or more of the 
respondents in one or more of the groups in the two NCHEMS surveys indicated a 
"Need To Know," Asterisks have been placed next to those measures receiving 
a high priority rating by each of the state-level groups surveyed, (In the 
survey of state-level officials there were 22 outcome measures that were en- 
dorsed by 50 percent of one or more groups as being of high priority, i.e,, 
"Need To Know,") 

ERIC 
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Figure 2 

Outcome Information Areas Contained in NCHEMS 
Outcome Measures Identification Study 



A, Student Knowledge and Skills Development - Information about student under- 
standingi competencies^ and attitudes relative to bodies of facts and 
principles and use of their intellectual and physical abilities. 

B. Student Educational Career Development - Information about student attitudes 
and success concerning certaTn academic pursuits (e.g»s student educational 
degree aspirations and attainments). 

C. Student Educational Satisfaction - Information that indicates the satisfaction 
of students about the knowledge and skills they have acquired and their prog- 
ress toward their educational and occupational career objectives* 

D, Student Occupational Career Development - Information about student attitudes 
and success concerning certain occupational goals and their job performance. 

Student Personal Development - Information about changes 1n students con- 
cerning the growth and maintenance of their personal life (e.g.p their 
ability to adapt to new situations ^ their self-concept, etc.)- 

F. Student Social/Cultural Development - Information about student abilities 
and attitudes in dealing with people and their interest in cultural 
activities. 

Community Educational Development ^ Information about the attitudes and 
success of nonmatriculating participants concerning the acquisition of 
knowledge and skills, personal and soda! development, and occupational 
career goals and performance. 

H, Cormunity Service - Information about the Impact of the opportunities and 
services pro^Tded by the institution and received by the community (e.g., 
agricultural extension services, cultural and recreational opportunities, 
etc, 

I. Community Impact - Information about the impact of an Institution's programs 
and its faculty, staff, and students (current and former) on the financial 
health, manpower supply, and attitudes of the cofranunity (local, state, 

or national) . 

J, Development of New Knowledge and Art - Information about new knowledge and 
art forms, created, applied, and reorganized as a result of an Institution's 
programs and Its faculty, staff, and students (current and former). 
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Figure 3 

Student Occupational Career Divtlopment Outcome Measures 



U Numbar and parcentaga of formar students (graduates and nongraduatas) 
survayad who ware employed within a certain time period after leaving 
the institution. 

2. Number and parcantaga of former students (graduates and nongraduatas) 
survayad who racaived the job of their first choice. 

3. Average first salary of former students. 

4. Distribution of former studants (graduates and nongraduatas) across 
income categories within a cartain tima period after leaving tha 
institution. 

5* Formar students (graduates and nongraduatas) scores on a scale measuring 
thair degree of satisfaction with their Job performance, 

6, Number of profassional occupation awards and citations racaived per 
formar student surveyed. 

7* Number and percentage of former students surveyed who are in management 
positions within a cartain time period after leaving the institution. 

8. Numbar of voluntary/involuntary changes in employment within a given 
time period par former student surveyed, : . 

9. Number of voluntary/involuntary changes in career field within a given 
time period per former student surveyed. 

10. Average first salary axpactations of students. 

11. Number and percentage of students who are aspiring to a particular 
type of occupational career. 

12. Number and parcantage of studants and/or formar studants surveyed who 
are saaking certain levels of employment. 

13. Number and parcentaga of former studants (graduates and nongraduatas) ^ 
surveyed accepting employment in their major field of study. 
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Figure 4 

Outcome Measures Endorsed by 601 
of One or More ''OMIS" Survey Groups as "Need To Know" 



1. Number of students passing certification or licensing exams (e,g., bar e^carn, 
CPA, LPN) on first attempt as a percentage of all students taking the axam, 

2. Student scores on tests that Indicate their ability to read, write, speak, 
and/or 1 isten. 

3. Number and percentage of students surveyed who have particlputed In activities 
that enhance their communication skills (e,g., debate, encounter groups, etc.). 

*4. Number and pereentnge of students surveyed identifying a certain degree, diploma, 
or certificate as the highest degree planned, 

5. Number and percentage of students surveyed who are taking noncredit, independent 
study, or special courses. 

*6. Number of students receiving a degree, diploma, or certificate within a certain 
time period, 

7. Average amount of time it takes a student to e^rn a degree, diploma, or 
certificate . 

*8. Number of students graduating from the institution after a certain period of 
time as a percentage of their entering class. 

9. Number and percentage of graduates for the year who transferred from another 
school 

*10. Number and percentage of students leaving the institution prior to receiving 
a degree, diploma, or certificate during a particular academic term or year. 

IL Student scores on a scale measuring their degree of satisfaction with their 
progress In achieving their educational career goals. 

12. Student scores on a scale measuring their degree of satisfaction with their 
progress in achieving their occupational career goals. 

*13. Number and percentage of former students (graduates and nongraduites) surveyed 
who were employed within a certain time period after leaving the institution. 

*14. Number and percentage of students surveyed who are aspiring to a particular 
type of occupational career. 

*15. Number and percentage of former students (graduates and nongraduates) surveyed 
accepting employnient in their major field of study. 

16, Number of nonmatrlcul ating participants enrolled 1n instructional programs as 
: _a percentage of the total number of persons In those programs. 

*17. Number and percentage of graduates of a particular graduating class who are 
employed in-state versus out-of=state. 

18. Community attitudes toward the institution (e.g., attitudes toward the insti- 
tution's contribution to community social/cultural activities and the 
institutions impact on the amount of crime In the community, 

19. Number of proposals funded for certain purposes (e.g., research versus training) 
by level of funding as a percentage of all proposals submitted. 

*20* Total dollar amount of gifts and/or grants received for certain purposes (e.g. , 
research versus training) as a percentage of total budget within a certain 
time period. 
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The results of the outcore measures identification studies do not completely 
Identify all of the outcome information valued by instlt'Utlon and state-level 
officials. They do Indicates however, that multiple indicators of performance 
end quality need to be considered in measuring program performance at the 
state level or at the institutional level, Furthermoret they confirm the fact 
that persons who have different decisionmaking responsibilities focus on and value 
certain kinds of outcomes. For instance, in both studies^ state-level officials 
consistently placed high Importance on Information and measures concerning 
Community Impact^ Community Service, and Student Occupational Career Development 
outcomes. As a result, 1t seems crucial for Institutional and state-level 
officials to consider the outcomes that are commonly and uniquely endorsed by 
each group so effective communication can be carried on about the desired and 
actual performance of institutions in program planning, budgeting, and evaluation 
situations. 



Problems Related to the Collection and Use of Outcome Information 
While most individuals concerned with postsecondary education recognize the need 
and urgency to use outcome Information for purposes of planning and budgeting, 
they are quick to point out the complexity of the problems associated with 
Identifying and measuring educational outcomes and incorporating this Information 
in the planning and budgeting process. 

One major difficulty has been that few explicit measures of program effectiveness 
have been available. Nor has much yet been done to show the links between resources 
and activities used and the attainment of desired outcomes even when these outcomes 
can be quantified. In short, 1t has been much easier to see what a program has 
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acconiplished In terms of activity or resource measures (e.g.i expenditures, 
student/ faculty ratios, enronment levels) than in terms of educational outcomes, 

A second problem often cited is that even when information about outcomes is 
available^ it is difficult to use since the techniques for analyzing and inter- 
preting these data are limited or are not well understood. For example, given 
all of the.-variables that potentially affect a particular outcome, it Is extremely 
difficult to determine cause-and-effect relationships, A further complexity 
results because many programs have joint outcomes. For example, a vocational- 
technical program may contribute to student knowledge and skill development 1n 
addition to producing various services to members of the business communijy, 

A third major difficulty is that motet planners and decision makers 
simply have a hard time translating their goals Into specific objectives stated 
in measurable outcome terms. Traditionally, goal setting is one of the first 
steps In the planning process; however, once the goals are stated, too often 
they remain in general ^ nonoperational terms* Because the goals lack translation 
Into specific, measurable descriptions, planners and decision makers have trouble 
utilizing them in selecting the optimal, or even promising^ courses of action 
and 1n evaluating the implemented programs* 

A fourth major difficulty 1s that the time span over which budgets are projected 
severely limits higher education's ability to assess the "educational value 
added" to students and society. In fact^ it would seem that the entire state 
planning and budgeting process is geared more toward preventing ,this type of 
assessment than to encourage 1t. As State Senator Robert Graham of Florida 
(1975) so aptly put 1t: ^ « 
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By appropriating either annual or bi-annual budgets^ as all fifty 
states do, every decision making level is able to -keep its options 
open' in the hopes that it will win approval the ne^ time around. 
At the same time there are enough Instances of reversals of previous 
commitments that no decision maker ever feels completely confident 
in proceeding with full program implementation. Instead, part of 
his resources must be preserved for the next annual budget fight. 

One outgrowth of all these problems is that the collection and use of outcome 
Information is often torn by the fear of potential misuses. One aspect of this 
fear 1s that the data will not portray an accurate picture of the actual results 
and benefits derived from an institution and its programs, A second and perhaps 
more basic concern, especially on the part of institutional administrators and 
faculty. Is the uncertainty about the ultimate findings and the actions that will 
be taken by persons in positions of control. This latter concern 1s based on the 
fear that the evaluation process will not adequately take into account those out- 
comes and benefits that are not quantifiable and those inputs and goals that are 
unique to a given institutional program. 

A second outgrowth Is the concern about how information should be presented for 
purposes of program evaluation and budget review. As State Senator Larry Borst 
of Indiana noted at a Legis 50 (nee Citizens Conference for State Legislators) 
Seminar last summer, "the situation that we In the legislature often encounter 
Is one that follows the "dump-truck theory'," That 1s, the legislature is provided 
tons and tons of data^ but considerably less useable Information. The problem, 
on the one hand, is the failure of the legislature to articulate clearly the 
precise Information 1t needs, and, on the other hand, is the institutions' fear 
that their situation will be misunderstood unless all the infortnatlon about their 
situation is provided. 

IB 
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Final Coninents 

The intent of this paper was to provide an overview of current practices and 
the problems that exist concerning the collection and use of higher education 
performance or outcome information for state-level planning and budgeting. 
The potential impact of the use of outcome information in the planning and 
budgeting has major implications for creating new Incentives and developing 
an environment for more effective use of resources and for improved educational 
programs that meet the needs of students and society. My major concern is that 
decision makers at the institutional and state levels join together in charting 
a positive and reasonable course of action that will help achieve those edu- 
cational outcomes that are valued by all of us* I hope that in this seminar 
we can begin getting together and start identifying what actions need to be 
taken In both the short-range as well as the long-range future. 



19 



18 



REFERENCES 



aerdahl , Robert. Criteria and Strategies for Program Discontinuance and 

Institutio nal Closure . A paper presented at the SHEEO Annual Conference, 
New Orleans, Louisiana, July, 1975. 

Chadwin, Mark L. "The Nature of Legislative Program Evaluation.'.' Evaluation, 
2 (2), 1975, pp. 45-49. " ' 

Graham, Robert. State Budget Reforni as a Way of Reforming Higher E ducation 
A paper presented at the EducationaT Commission of the States Seminar 
on Making Decisions in a Time of Fiscal Stringency," Denver, Colorado, 
December, 1975, 

Glenny, Lyman, et al . State Budgeting for Hioher Education : Data Digest 
Berkeley, California: Center for Research and Deveiopment in Higher 
Education, 1975. ^ 

Folger,John K. Allocation of State Funds on a Performance Effectiv eness 

Criterion. A final proposal submitted by the Jennessee Higher Educa 1 1 on 

Commission to the Fund for the Improvement of Postsecondary Education, 
DHEW, March, 1975. 



Meisinger, Richard J., Jr., Purves, Ralph A., and Schmldtlein, Frank A. 
"Productivity From An Interorganizational Perspective." In R.A. 
Wallhaus (ed.) Measuring and Increasing Academic Productivit y. 
San Francisco: Jossey-Bass, Inc., 1975. 

Micek, Sidney S. and Wallhaus, Robert A. An Introduction to the Identification 
and Use of Higher Education Qutcome Information . Boulder. Colorado: 
Western Interstate Commission for Higher Education, 1973. 

Micek, Sidney S. and Arney, William R. The Higher Education Qutcome Measures 
Identification Study: A Descriptive Summary . BouTder, Colorado: " 
Western Interstate Commission for Higher Education, 1974. 

Micek, Sidney S. and Oberbeck, Del ma. State-Level Higher Education Outcome 
Measures Identification Study . Boulder, Colorado: Western Interstate 
Commission for Higher Education, forthcoming, 1976. 

Smith, Donald K. Academic Program Audit and Review as a Means of Resource 
Allocation. A paper presented at the Educational Commission of the 
States Seminar on "Making Decisions in a Time of Fiscal Stringency," 
Denver, Colorado, December, 1975. 



20 



